Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11806

Spark 2.0 deprecations and removals

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Umbrella
    • Status: Resolved
    • Major
    • Resolution: Done
    • None
    • 2.0.0
    • Spark Core

    Description

      This is an umbrella ticket to track things we are deprecating and removing in Spark 2.0.

      Attachments

        Issue Links

        relates to

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-12169 SparkR 2.0

        • Major - Major loss of function.
        • Resolved
        1.
        		 Remove support for Hadoop < 2.2 (i.e. Hadoop 1 and 2.0) Sub-task Resolved Reynold Xin Actions
        2.
        		 Remove Bagel Sub-task Resolved Reynold Xin Actions
        3.
        		 Remove Bagel Sub-task Resolved Reynold Xin Actions
        4.
        		 Remove Guava's "Optional" from public API Sub-task Resolved Sean R. Owen Actions
        5.
        		 SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T] Sub-task Resolved Reynold Xin Actions
        6.
        		 Remove Mesos fine-grained mode subject to discussions Sub-task Resolved Michael Gummelt Actions
        7.
        		 Remove YARN Client / ClientArguments Sub-task Resolved Saisai Shao Actions
        8.
        		 Remove history serving functionality from standalone Master Sub-task Resolved Bryan Cutler Actions
        9.
        		 Remove closure serializer Sub-task Resolved Sean R. Owen Actions
        10.
        		 Java mapPartitions Iterator->Iterable is inconsistent with Scala's Iterator->Iterator Sub-task Resolved Sean R. Owen Actions
        11.
        		 Remove usage of Hadoop deprecated APIs and reflection that supported 1.x Sub-task Resolved Sean R. Owen Actions
        12.
        		 Rename "dynamic allocation" to "elastic scaling" Sub-task Closed Andrew Or Actions
        13.
        		 Remove JobLogger Sub-task Resolved Reynold Xin Actions
        14.
        		 Remove HTTPBroadcast Sub-task Resolved Reynold Xin Actions
        15.
        		 Remove TTL-based metadata cleaning (spark.cleaner.ttl) Sub-task Resolved Josh Rosen Actions
        16.
        		 Confusing behavior of fold function of RDD in pyspark Sub-task Resolved Sean R. Owen Actions
        17.
        		 Remove deprecated methods in SQL / DataFrames Sub-task Resolved Reynold Xin Actions
        18.
        		 Java count(AprroxDistinct)ByKey methods return Scala Long not Java Sub-task Resolved Sean R. Owen Actions
        19.
        		 Remove some deprecated APIs in RDD/SparkContext Sub-task Resolved Reynold Xin Actions
        20.
        		 Remove SparkContext preferred locations constructor Sub-task Resolved Reynold Xin Actions
        21.
        		 Clean up build warnings: 2.0.0 edition Sub-task Resolved Sean R. Owen Actions
        22.
        		 Remove unused code related to Hadoop 0.23 Sub-task Resolved Kousuke Saruta Actions
        23.
        		 Remove Vector, VectorSuite and GraphKryoRegistrator which are deprecated and no longer used Sub-task Resolved Kousuke Saruta Actions
        24.
        		 Remove block manager's internal "external block store" API Sub-task Resolved Reynold Xin Actions
        25.
        		 Remove submit python/R scripts through pyspark/sparkR Sub-task Resolved Jeff Zhang Actions
        26.
        		 Remove HistoryServer old multiple files format Sub-task Resolved Felix Cheung Actions
        27.
        		 Deprecate TaskContext.isRunningLocally() Sub-task Resolved Josh Rosen Actions
        28.
        		 Change default build to Hadoop 2.7 Sub-task Resolved Unassigned Actions
        29.
        		 Follow up SPARK-12707, Update documentation and other related code Sub-task Resolved Unassigned Actions
        30.
        		 Spark on Yarn remove deprecated configs for 2.0 Sub-task Closed Boyang Jerry Peng Actions
        31.
        		 Remove usage of ADD_FILES in pyspark Sub-task Resolved Jeff Zhang Actions
        32.
        		 Remove deprecate APIs from GraphX Sub-task Resolved Takeshi Yamamuro Actions
        33.
        		 Deprecate "yarn-client" and "yarn-cluster" Sub-task Resolved Saisai Shao Actions
        34.
        		 Logging: Make it either stable or private[spark] Sub-task Resolved Sean R. Owen Actions
        35.
        		 Move UserDefinedFunction into sql.expressions package Sub-task Resolved Reynold Xin Actions
        36.
        		 Upgrade to Jersey 2 Sub-task Resolved Matt Cheah Actions
        37.
        		 Remove SparkContext.metricsSystem Sub-task Resolved Reynold Xin Actions
        38.
        		 Remove the support of SIMR cluster manager Sub-task Resolved Saisai Shao Actions
        39.
        		 Remove reference to Tachyon in cluster & release script Sub-task Resolved Reynold Xin Actions
        40.
        		 Move network/* modules into common/network-* Sub-task Resolved Reynold Xin Actions
        41.
        		 Move tags and unsafe modules into common Sub-task Resolved Reynold Xin Actions
        42.
        		 Move docker, extras modules into external Sub-task Resolved Sean R. Owen Actions
        43.
        		 Move misc top-level build files into appropriate subdirs Sub-task Resolved Sean R. Owen Actions
        44.
        		 remove typed operations (map, flatMap, mapPartitions) from Python DataFrame Sub-task Resolved Wenchen Fan Actions
        45.
        		 Move streaming-flume, streaming-mqtt, streaming-zeromq, streaming-akka, streaming-twitter to Spark packages Sub-task Resolved Shixiong Zhu Actions
        46.
        		 Move docs of streaming-mqtt, streaming-zeromq, streaming-akka, streaming-twitter to Spark packages Sub-task Resolved Shixiong Zhu Actions
        47.
        		 Consider removing Kafka modules from Spark / Spark Streaming Sub-task Closed Unassigned Actions
        48.
        		 Remove Streaming Akka docs from Spark Sub-task Resolved Shixiong Zhu Actions
        49.
        		 Move streaming-flume back to Spark Sub-task Resolved Shixiong Zhu Actions
        50.
        		 Remove straggler references to Tachyon Sub-task Resolved Liwei Lin(Inactive) Actions
        51.
        		 Require Java 8 for Spark 2.x Sub-task Closed Unassigned Actions
        52.
        		 Change SparkListener from a trait to an abstract class, and remove JavaSparkListener Sub-task Resolved Reynold Xin Actions
        53.
        		 Remove HashShuffleManager Sub-task Resolved Reynold Xin Actions
        54.
        		 Put examples packaging on a diet Sub-task Resolved Marcelo Masiero Vanzin Actions
        55.
        		 Java sampleByKey methods take ju.Map but with Scala Double values; results in type Object Sub-task Resolved Sean R. Owen Actions
        56.
        		 Remove support for "zk://" master URL Sub-task Resolved Sean R. Owen Actions
        57.
        		 Add a deprecation warning for Python 2.6 Sub-task Resolved Unassigned Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rxin Reynold Xin
            rxin Reynold Xin
            Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment